Running Header: Comparative Validity of the DTMS and SAT-M 



Comparative validity of the 
Descriptive Tests of Mathematical Skills (DTMS) 
And SAT - Mathematics (SAT-M) 
for Predicting Performance in 
Freshman College Mathematics Courses: Prefatory 

Report. 



M. Padraig M. M. McLoughlin 
Morehouse College 
pmclough @ morehouse.edu 

Dontrell A. Bluford 
Morehouse College 
blumagic29@yahoo.com 



Paper presented at the annual meeting of the Eastern Educational Research Association, 

Clearwater, FL 
12 February 2004. 




Comparative Validity of the DTMS and SAT-M, page 1 



Abstract 

Comparative validity of the Descriptive Tests of Mathematical S kil ls (DTMS) 
And SAT - Mathematics (SAT-M) for Predicting Performance in 
Freshman College Mathematics Courses. 

M. Padraig M. M. McLoughlin & Dontr e l l A. Bluford 
Morehouse College 



This study investigated the predictive validity of the Descriptive Tests of 
Mathematical Skills (DTMS) and the SAT -Mathematics (SAT-M) tests as placement 
tools for entering students in a small, liberal arts, historically black institution (HBI) using 
regression analysis. The placement schema is four-tiered: for a remedial algebra course, 
college algebra course, pre-calculus course, and a first calculus course. Low psychometric 
indices were found between score earned on the Elementary Algebra Skills (EAS), 
Intermediate Algebra Skills (IAS), and Calculus Readiness: Functions and Graphs (CR) of 
the DTMS and grade earned in College Algebra, Pre-calculus, and Calculus I, respectively, 
and between score earned on the SAT-M and grade earned in College Algebra, Pre- 
calculus, and Calculus I. 

Define E denote the score earned on the EAS, I the score earned on the IAS, R the 
score earned on the CR, S the score earned on the SAT-M, B the grade earned in the Basic 
Mathematics course, A the grade earned in the College Algebra course, P the grade earned 
in the Pre-calculus class, and C the grade earned in the Calculus I class. The models for the 
DTMS produced were as follows: A = 1.504 + 0.039E for the EAS of the DTMS; 

P = 1.030 + 0.0841 for the IAS of the DTMS; and, C = 1.670 +0.050R for the CR of the 
DTMS; whereas, the models for the SAT-M produced were: A = 0.241 + 0.004S; 

P = 0.695 + 0.003S; C = -0.519 + 0.005S. 

Perhaps the most interesting component of the study was not the regression equations, 
correlations, or any other parametric statistical portion of the study but the effect of the 
change of policy on the placement schema at the college. If placement decision were based 
on scores of SAT-M rather than on scores earned on the DTMS, there seems to be a shift of 
students such that less were placed in the remedial class and more into the courses that 
allowed a student to earn college credit and satisfy core curriculum requirements for 
mathematics. 

The paper ends with recommendations for further research on minimum competence 
testing and placement and notes the follow-up studies to this study that will be done by the 
researchers. 




Comparative Validity of the DTMS and SAT-M, page 2 



INTRODUCTION 

The purpose of the study was to add to the body of evidence regarding the 
predictive validity of three tests of the Descriptive Tests of Mathematical Skills (DTMS), 
the Elementary Algebra Skills (EAS) test, Intermediate Algebra Skills (IAS) test, and 
Calculus Readiness: Functions and Graphs (CR) test or of the SAT - Mathematics 
(SAT-M) as tools used to place students in freshman level mathematics courses. This 
study investigated the relationship between students’ scores earned on the EAS, IAS, and 
CR of the DTMS and the SAT-M and grades earned in a College Algebra, Pre-calculus, 
and Calculus I class (respectively) at a four year, private, liberal arts, and historically black 
college in the southern United States. This is an attempt to estimate the predictive validity 
of the EAS of the DTMS versus the SAT-M and grades earned in the College Algebra 
course, the IAS of the DTMS versus the SAT-M and grades earned in the Pre-calculus 
course, and the CR of the DTMS versus the SAT-M and grades earned in the Calculus I. 
The student body at the four year, private, liberal arts, and historically black college was all 
male of which approximately 97.359% 1 were African-American. The overall enrolment at 
the college was approximately 2,800 students. 

At the liberal arts college, the mathematics department’s placement was based on 
the results of a student's performance on three (EAS, IAS, and CR) of the four components 
of the DTMS through 1998. In 1999, the placement scheme was changed to placement in 
freshman mathematics courses based on results from the SAT-M. The policy was changed 
through the Office of Academic Affairs at the college; but, faculty in the mathematics 
department reported anecdotal evidence to suggest that more students were misplaced 
(mostly at a higher level than prepared for) using the scheme of the SAT-M rather than the 
scheme of the DTMS. The college requires all students compleat a minimum of six credit 
hours of course- work in mathematics at the 100-level or above (there is a more stringent 
requirement for students majoring in the ‘hard’ sciences than the social sciences or 
humanities). 



1 Note: all constants reported are precise to two significant figures. 
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The Descriptive Tests of Mathematical Skills (DTMS) of the College Board 
(Educational Testing Service, 1979; revised, 1988) includes four tests: Basic Arithmetic 
Skills (BAS), Elementary Algebra Skills (EAS), Intermediate Algebra Skills (IAS), and 
Calculus Readiness: Functions and Graphs (CR). Each is in a four-option multiple choice 
format, and the maximum testing time for each is 30 minutes. The BAS and EAS tests are 
35-item tests. The IAS and CR are 30-item tests. These tests can be used separately or in 
any combination to conduct minimum mathematics competency placement of students for 
many types of courses including, but not limited to, freshman level mathematics courses. 

The 35 question EAS is comprised of nine questions covering operations with real 
numbers, nine questions covering operations with basic algebraic expressions, eight 
questions covering solutions of basic equations and inequalities, and nine questions 
covering applications of the aforementioned. The 30 question IAS is comprised of seven 
questions covering algebraic operations, eight questions covering solutions of equations 
and inequalities, eight questions covering geometry, and seven questions covering 
applications in statistics, probability, and data interpretation. The 30 question CR is 
comprised of eight questions covering algebraic functions, nine questions covering 
exponential and logarithmic functions, and thirteen questions covering trigonometric 
functions (College Entrance Examination Board, 1989; Educational Testing Service, 1988). 

The SAT - Mathematics (SAT-M) is in a four-option or five-option multiple 
choice format 60 question test administered in three part (two 30 minute parts and one 15 
minute part) and is comprised of approximately three-tenths of the 60 questions cover 
Arithmetic (Number and Operations), approximately three-tenths of the 60 questions 
covers Algebra and Functions, approximately three-tenths of the 60 questions cover 
Geometry and Measurement, and approximately one-tenth covers Data Analysis, Statistics, 
and Probability. 

There is scant research regarding the use of the DTMS in minimum competency 
placement testing programmes designed to validate the mathematical skills of entering 
freshmen students into college. Many of the instruments used for measuring mathematical 
skills of the entering freshmen are locally developed, but a few are standardised tests (most 
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oft the BAS, EAS, IAS, or CR of the DTMS, the ACT-M (American College Testing 
[Programme] Mathematics), the SAT-M, or the SATII-M (SATII Mathematics). 

Prior researchers estimated the predictive validity of the Elementary Algebra Skills 
(EAS) test of the DTMS, other tests of the DTMS, or combinations of the aforementioned. 
For example, Bridgeman (1980) found the DTMS was “predictive of end of course grades 
when administered at the beginning of the course” (page 82). Correlation between the 
score earned on the EAS and grade in a college algebra course ranged from a low of 0.30 to 
a high of 0.58 for 10 colleges or university with sample sizes ranging from 16 to 99 
subjects. Bridgeman (1982) compared the validity of the EAS and LAS to the Scholastic 
Aptitude Test - Mathematics (SAT-M) and found the EAS or IAS a better predictor of 
performance for a remedial algebra course than the SAT-M, whereas the EAS, LAS, and 
SAT-M performed about equally well for predicting performance in a calculus course. He 
found psychometric indices of 0.46 and 0.47 between the grade earned in an Elementary 
Algebra course and the score earned on the EAS based on 73 subjects at a small private 2- 
year college and on 198 subjects at a large public 4-year college, respectively. Suddick and 
Collins (1985) found a test-retest reliability coefficient of 0.72 for the EAS of the DTMS 
for older, adult transfer students using a small sample. Suddick and Collins (1982) also 
investigated the predictive validity of the EAS and IAS for grades earned in a calculus or 
statistics course for older upper division students. They found that students who 
passed either the EAS or IAS performed significantly better than did those who failed the 
DTMS, were remediated, and then earned a grade in either the statistics or calculus course. 
Suddick and Collins (1984) further investigated the predictive validity of the EAS for 
grades earned in a calculus course for older Master’s level students. Suddick and Vaccarro 
(1983) discussed the issues involved in minimum competency testing using the BAS for 
business students. Dwinell (1985) investigated the predictive validity of the BAS and EAS 
to estimate exit from a remedial mathematics course for students who earned failing scores 
on the Basic Skills Examination - Mathematics test and found the EAS contributed the 
“most variance to an estimate of prediction of exit from [remedial] math courses” (page 



12 ). 
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Nonetheless, all of the aforementioned research reports used the original version of the 
DTMS as an instrument and only the original Bridgeman studies (1980, 1981, and 1982) 
considered the CR of the DTMS. The DTMS was revised in 1988, and very little research 
exists that investigated the predictive validity of the revised DTMS. Meyer, Woodard, and 
Suddick (1994) investigated the predictive validity of the BAS and EAS (revised DTMS) 
for grades earned in an advanced mathematics concepts and structure course for upper- 
division elementary education students and found a moderate predictive validity coefficient 
(R = 0.49) using a small sample. 

McLoughlin (1999, 2000) found that students who passed the EAS, LAS, and CR of 
the DTMS did have a significantly higher average level of performance in College Algebra, 
Pre-calculus, and Calculus I, respectively, than did the students who did not have the 
prerequisite minimum competency skills in mathematics as measured by the EAS, IAS, or 
CR of the DTMS but were successfully remediated. He also opined that the difference 
between the grades earned in each of the respective courses was practically significant. His 
findings corresponded with the findings of Bridgeman (1980, 1981, and 1982) for the 
original version of the DTMS. 

Bridgeman and Wendler (1989) investigated the predictive validity of the SAT- M by 
collecting grades from freshman mathematics courses at 10 colleges (3,499 students). They 
compared the SAT-M to tests that were specifically designed for placement purposes and 
found the SAT-M score was a relatively poor predictor of grades in most courses. 

However, they found that the SAT-M significantly improved predictions for freshman 
mathematics course grades (especially for calculus courses) over using just high school 
grade point average (GPA). 

Notwithstanding, all of the studies heretofore mentioned have been conducted at 
colleges or universities that are not analogous to the college where the data was collected in 
this study. 2 In addition, some of the studies did not account for a percentage of African- 
Americans in the sample, while others had over 90% White subjects; ergo, the possibility 
exists that the predictive models from those studies may not generalise to the population 

2 It should be noted that the DTMS was piloted at the college where the data was collected for this study in 
the late 1970s and a member of the faculty at said college (no longer a member of the faculty) was on the 
team that designed the DTMS. 
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studied in this project. Notwithstanding, this research assumes that the studies previously 
noted are valid and, therefore, the research question in this study includes whether or not 
the models from previous studies of the EAS, IAS, and CR of the DTMS and the SAT-M 
are comparable to the model generated in this study. 

METHODS 

This paper estimates predictive validity (overall significance level of 0.05) to model 
prediction of grade earned in the respective college mathematics course using the score 
earned on the EAS, IAS, CR, and SAT-M. Each of the EAS, IAS, and CR were compared 
to the placement method used for the SAT-M. The researchers used regression to analyse 
the data. 

The researchers used regression analysis (a = 0.05) to analyse the difference 
between the means of the grades earned by students in the course he was placed of those 
who passed and those who failed the part of the DTMS but were placed in the course 
previous and then took the course the following semestre to build a predictive model. It 
was determined a stable and interpretable model was produced that only used the score 
earned on the EAS, IAS, CR, or SAT-M as a regressor with the grade earned in the 
respective freshman-level course (College Algebra, Pre-calculus, or Calculus I) as the 
measured response rather than for each part of the DTMS scores or SAT-M along with a 
dummy variable indicating pass or fail status and an interaction term as the regressors. 

A small pilot study (n = 47) was conducted in the spring of 1997 using scores earned 
on the EAS, IAS, and CR of the DTMS and grade earned in College Algebra, Pre-calculus, 
and Calculus I from fall of 1993 through fall of 1994 for a random sample of entering 
freshmen in the fall of 1993 to 1) determine an adequate period of grade records needed to 
ensure that enough time was allowed for those who placed in the remedial mathematics 
course to successfully compleat the course and be able to attempt to earn a grade in College 
Algebra (to be discussed in the next paragraph); and, 2) test underlying statistical 
assumptions (normally independent distribution, underlying continuity, linearity, and 
homoscedasity) for conducting a regression analysis. The underlying assumptions were 
found to be tenable based on the results of the pilot study. 
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A record of performance on the EAS, IAS, and CR of the DTMS for the 186 
subjects was obtained as well a record of performance on the SAT-M. The scores earned 
on the EAS were integer values from 0 to 38, inclusive; scores earned on the IAS or CR 
were integer values from 0 to 30, inclusive, but were considered continuous for the study. 
The scores earned on the SAT-M were integer values from 200 to 800 (multiples of ten), 
but were considered continuous for the study. Also, mathematics departmental records of 
grades earned by students in the freshman-level courses for the period of fall 1998 through 
spring 2000, inclusive, were used. The grade records period of fall 1998 through spring 
2000 was used because by using these parametres maximisation of the usable number of 
records in the study was possible. Some students defer enrolling in their mathematics 
courses at the college for well over a year, therefore, 1998 - 2000 covers a reasonable time 
for entering freshman-level courses and earning a grade and yet restricts the time for 
entering freshman-level courses so that the predictive power of the DTMS was not 
compromised. 

The grade earned in freshman-level courses is based on the 4-point plus - minus scale. 
The grade was considered a continuous variable. Only the grade a subject earned on his 
first attempt in a freshman-level course was considered. The definition of earned grade 
used in this study was an “A+” (4.0) to an “F” (0.0). 3 The grades “W,” “WP,” and “WF” 
which signify withdrawal, withdrawal passing, and withdrawal failing, respectively, were 
not considered as an earned grade in the course. Thus, if a “W” was assigned for subject 
X, for example, in the fall of 1998, but he earned a “D” in the spring of 1999, then a 1.0 
(the numeric equivalent for a “D”) was recorded for Subject X. The grades were recorded 
twice independently, then the two files were compared with the departmental records, and a 
final data set was thus produced. 

Cut scores for the three DTMS tests were as follows: 24.5 for the EAS, 20.5 for the 
IAS, and 20.5 for the CR. Ergo, a student who selects 24 correct answers or less on the 
EAS is required to enrol in a 0 semestre credit hour remedial mathematics course (which is 
a review of high school algebra). The student was required to earn a "C" or better in the 



3 An “A+” is recorded as 4.0, an “A” is recorded as 4.0, a “A-” is recorded as 3.7, a “B+” is recorded as 3.3, a 
“B” is recorded as 3.0, a “B-” is recorded as 2.7, a “C+” is recorded as 2.3, a “C” is recorded as 2.0, a “C-” 
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course in order to enrol in College Algebra. A student who selects 25 correct answers or 
more was permitted to enrol in College Algebra. Further, if a student selects 20 correct 
answers or less on the IAS, then he was required to enrol in College Algebra, but a student 
who selects 21 correct answers or more on the IAS was permitted to enrol in Pre-calculus, 
and if he selects 21 correct answers or more on the CR test, he was permitted to enrol in 
Calculus I. 

The decision schema adopted by the college in 1999 for use with the SAT-M is as 
follows: cut scores for the SAT-M were as follows: 415, 535, and 595. Hence, a student 
whose score is less than 410 is required to enrol in a 0 semestre credit hour remedial 
mathematics course; a student whose score is greater than or equal to 420 but less than 540 
is permitted to enrol in College Algebra; a student whose score is greater than or equal to 
540 but less than 600 is permitted to enrol in Pre-calculus, and a student whose score is 
greater than or equal to 600 is permitted to enrol in Calculus I. 

Simple regression analysis was performed to ascertain the validity of the placement 
model using the EAS, IAS, and CR of the DTMS for this population. The grades earned in 
the freshman-level courses were assumed independent. Residual analyses were performed 
which confirmed the tenability of the assumptions necessary for inference in regression 
analysis; namely, that the dependent variable, grade earned in the respective freshman-level 
course, was approximately normally distributed, the relationship between the independent 
and dependent variables was linear, and that there was homogeneity of variances. 

The study was also to have included another component which has not been executed 
as of this time. SAT-M records for freshmen who entered the college in the fall of 1999 
were to be used along with mathematics departmental records of grades earned by students 
in the freshman-level courses for the period of fall 1999 through spring 2001. A model was 
to be produced using SAT-M as a regressor with the grade earned in the respective 
freshman-level course (College Algebra, Pre-calculus, or Calculus I) as the measured 
response. This model was to be compared to the 1998 model for the SAT-M as well as for 
the DTMS for the 1998 data. However, the researchers have not gained access to the 1999 
SAT-M data; thus, this study is a prefatory report. Therefore, herein for subjects and results 



is recorded as 1.7, a “D+” is recorded as 1.3, a “D” is recorded as 1.0, a “D-” is recorded as 0.7, and an “F” is 
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sections a detailed mention of the 1999 SAT-M portion of the study will not be included; 
but, in the discussion section the 1999 portion will be mentioned. 



SUBJECTS AND SAMPLING TECHNIQUE 

The subjects were a subset of the set of entering freshmen at four year, private, 
liberal arts, and historically black college in the fall of 1998. Approximately 97% of the 
entering freshmen of the class of 1998 were United States citizens, with the remainder 
from the Caribbean or Africa. The placement testing policy at the college required all 
entering freshmen to take three DTMS tests : EAS, LAS, and CR. Cut scores for the three 
DTMS tests were as follows: 24.5 for the EAS, 20.5 for the IAS, and 20.5 for the CR. 

Hence, the subjects of the study are the subset (n = 186) of the entering freshmen 
(N = 599) who entered the college in the fall of 1998. All 599 entering freshmen took the 
three DTMS tests: EAS, LAS, and CR and the 186 subjects of this study were randomly 
selected from the 599 entering freshmen using a modified four stage systematic sampling 
design created by the primary author. 

The systematic sampling procedure was as follows. For stage one, one hundred 
eighty-six numbers were pseudo-randomly selected from N599. 4 If the subject also reported 

SAT-M scores then the record was used. If not, then stage two was employed which 
started with selecting a pseudo-random number from Zio\{0}. 5 If that record was not used 

before and also contained a reported SAT-M score, then it was used; if not, then stage three 
was employed which required selecting a pseudo-random number was used from Z 5 \{0}. 6 

If that record was not used before and also contained a reported SAT-M score, then it was 



recorded as 0.0. The grades of “F+” and “F- ” do not exist. 

4 N599 = {x I x eN a x < 600}. 

5 Z 10 \{0}= {-10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}. 
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used; if not, then stage four would be employed a pseudo-random number was used from 
Z3\{0}. 7 The pseudo-random number tables were generated with SPSS 10.0 (without a 

pre-set seed) and it should be noted that only through stage three of the sampling process 
was needed. 

The result was four subsets were created composed of those who failed the EAS, 
IAS and CR (group 0, n 0 = 46); those who passed the EAS but failed the IAS and CR 
(group 1, ni = 66); those who passed the EAS and IAS but failed the CR (group 2, 
n 2 = 38); and, those who passed the EAS, IAS, and CR (group 3, n 3 = 24). Group 0 
members were placed in the remedial Basic Mathematics course, Group 1 members were 
placed in College Algebra, Group 2 members were placed in Pre-calculus, and group 4 
members were placed in Calculus I. 

RESULTS 

Let E denote the score earned on the EAS, I denote the score earned on the IAS, R 
denote the score earned on the CR, S denote the score earned on the SAT-M, B denote the 
grade earned in the Basic Mathematics course, A denote the grade earned in the College 
Algebra course, grade earned in the Pre-calculus class, and C denote the grade earned in 
the Calculus I class. 

KR 20 reliability of each of the tests of the DTMS were computed (such was not the 
case for the SAT-M since the item responses were not available). KR 2 o reliability of the 
EAS instrument was 0.8744; KR 2 o reliability of the IAS instrument was 0.8443; and, 

KR 20 reliability of the CR instrument was 0.8769. 

Descriptive statistics for each of the tests were computed for all entering freshmen in 
the fall of 1998 (N = 599). For the EAS, pe = 25.46, Ge « 6.164, minimum score 0, 
maximum score 35; for the IAS, pi = 17.51, gi « 5.571, minimum score 0, maximum score 
30; for the CR, Pr = 12.61, gr « 6.589, minimum score 0, maximum score 29; and, for the 
SAT-M (Ns = 510), ps = 524.71, os « 85.15, minimum score 350, maximum score 780. 



6 Z 5 \{0}= {-5, -4, -3, -2, -1, 1, 2, 3, 4, 5}. 
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The assumption of normality was tenable because the normal probability plot using 
jack-knife residuals for a linear model fitted to the data showed no obvious deviation 
from normalcy (see figure 1 which illustrates the EAS - CA relationship). The assumption 
of homogeneity of variances was tenable because analysis of predicted- A versus the jack- 
knife residual showed no blatant departure from the assumption. 




Observed Cum Prob 

Figure 1: Normal Probability Plot Using Jack-knife for EAS and CA. 



The models for the DTMS (all of the form DV = (3o + Pi (IV) + s => D = p 0 + Pi (IV)) 
produced were as follows: 

A = 1.504 + 0.039E for the EAS of the DTMS such that f * 0.146, SE(p 0 ) * 0.608 and 
SE(pO * 0.026; P = 1.030 + 0.0841 for the IAS of the DTMS such that r « 0.299, 

SE(po) * 0.523 and SE(pj) * 0.029; and, C = 1.670 +0.050R for the CR of the DTMS such 
that r« 0.227, SE(p 0 ) « 0.592 and SE(pi) * 0.031. 

The models for the SAT-M (modelling if it were used as a predictor of performance in 
the classes) produced were as follows: 

A = 0.241 + 0.004S such that r* 0.194, SE(p 0 ) « 1.116 and SE(pi) * 0.002; 

P = 0.695 + 0.003S such that r« 0.205, SE(p 0 ) « 0.695 and SE(p0 * 0.002; 
and, C = -0.519 + 0.005S such that r« 0.341, SE(p 0 ) « 1.303 and SE(pi) « 0.002. 



7 Z 3 \{0}={-3, -2, -1,1,2, 3}. 
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The respective mean grades earned per category of the EAS, IAS, and CR of the 
DTMS are reported in table 1 and for the SAT-M the mean grades earned per category are 
reported in table 2. 





Basic Math 
Math 090 


College Algebra 
Math 100 


Pre-Calculus 
Math 120 


Calculus I 
Math 161 


Failed EAS (E < 24) 


|LL = 2.3404 

5 =1.387 
n = 47 


JLI = 2.2611 

5 =1.393 
n-36 


JLI = 1.900 
5 =1.159 
n=17 




Passed EAS (25 < E) 
Failed IAS (I < 20) 




JLI = 2.4743 

a =1.129 
n=70 


|LL = 2.4667 
5 =1,168 
n=36 


JLI = 1.9661 

5 =1.397 
n=8 


Passed EAS (25 < E) 
Passed IAS (21 < I) 
Failed CR (R < 20) 






|LL = 2.8353 

5 =1.039 
n=34 


JLI = 2.7278 1 

5 =1.037 
n=18 | 


Passed EAS, 

IAS, & CR (25 < E, 
21 <1, a 21 <R) 








JLI = 2.6652 
a =1.356 
n=23 



Table 1: 1998 Sample DTMS Mean Grades, Standard Deviation, and Sample Sizes 





Basic Math 
Math 090 


College Algebra 
Math 100 


Pre-Calculus 
Math 120 


Calculus I 
Math 161 


Level 0 
S < 410 


|i = 1.383 
6 =1.744 
n = 6 


(1 = 2.800 

6 =1.377 
n=5 


|i = 3.000 
6 =1.4142 
n=2 




Level 1 
410 < S < 540 


JLL = 2.511 

5 =1.271 
n = 38 


|LL = 2.2250 

5 =1.229 
n=72 


JLI = 2.2851 

S =1.220 
n=47 


JLI = 2.1000 

5 =1.442 
n=14 


Level 2 
540 < S < 600 


|i =3.000 

d DNE 8 
n = 1 


|LL = 2.927 

a =1.073 
n=22 


JLL = 2.6091 

a =1.338 
n=22 


JLI =2.1121 

a =1.119 
n= 11 






JLI = 3.300 


JLL = 3.0636 


JLI = 3.0700 



DNE denotes does not exist. 
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Level 3 




a DNE 


a =0.825 


a =1.389 


X > 600 




n = 1 


n=ll 


n=20 



Table 2: 1998 Sample SAT-M Mean Grades, Standard Deviation, and Sample Sizes 



There was not a significant difference in the means for grade earned in any class by 
any test except for grade earned in Calculus I with respect to the SAT-M, (t « -2.1 12; df= 
29; p < 0.05) and none of the predictive validity coefficients were significant. 

DISCUSSION 

Students having the prerequisite minimum competency skills in mathematics as 
measured by the EAS, IAS, and CR of the DTMS did not have a significantly higher 
average level of performance in the respective courses (College Algebra, Pre-calculus, and 
Calculus I) than did the students who did not have the prerequisite minimum competency 
skills in mathematics as measured by the EAS, IAS, or CR of the DTMS and were 
successfully remediated. This lack of significant difference between the grades earned 
conflicts with the findings of McLoughlin (2000). Note that in the 2000 study, all of the 
grades earned in the respective courses (College Algebra, Pre-calculus, and Calculus I) 
were statistically and practically different (see table 3). 





Basic Math 
Math 090 


College Algebra 
Math 100 


Pre-Calculus 
Math 120 


Calculus I 
Math 161 


Failed EAS 


JLL = 1.877 
6 =1.357 
n = 178 


ILL = 1.231 
0=1.159 
n=139 


JLL = 1.290 
a =1.198 
n=69 


JLL = 1.618 ! 

a =1.398 
n=I I 


Passed EAS 
Failed IAS 




|i =2.008 

ct =1.268 
n=214 


JLL = 1.952 
a =1.255 
n=150 


jlL = 1.561 1 

5 =1.128 
n=54 


Passed IAS 
Failed CR 




{i =2.731 
o =1.230 
n=29 


(I = 2.462 

0=1.198 

n=116 


(I = 2.024 

5 =1.313 
n=67 


Passed EAS, 
IAS, & CR 








JLL = 2.986 1 

a =1.167 
n=50 



Table 3: DTMS Mean Grades 1994 Census (2 significant digit accuracy), 
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Standard Deviations (2 significant digit accuracy), and Sample Sizes 



There are several possible explanations for this difference in findings. First, the 2000 
study used all of the data (entering freshmen in the fall of 1994) and hence the finding 
contained within this study could be due to sampling. Second, there was a practical change 
in policy at the college between the two studies. The last day to drop a course without 
penalty was changed in the intervening years so that those subjects in the 2000 study had to 
withdraw from a course by the week after mid-term of a semestre whilst the subjects in this 
study had until one week before the end of the semestre to withdraw from a course. This 
policy change may also explain the difference in the mean grades earned per course 
between and betwixt the two data sets. It could be that since students who were failing a 
course had approximately six more weeks to decide to drop a course without penalty versus 
students in the 2000 study there might have been more students withdrawing than in the 
previous study. Clearly the mean grades earned are substantially higher in this data set 
than in the 2000 data set (except for mean grade earned in Basic Mathematics (the remedial 
course)). Third, the student population entering the college in 1998 may be more 
mathematically adept than those who entered the college in 1994. 

Another interesting finding is that in the College Algebra model and the Calculus I 
model the correlation between the SAT-M score and the grade earned was higher between 
A and SAT-M (?« 0.194) and between C and SAT-M (?« 0.341) than between the score 
earned on the EAS and grade earned in College Algebra (r» 0.146) and between the score 
earned on the CR and grade earned in Calculus I (?« 0.227); only for the IAS was this not 
so (r « 0.299 between IAS and P as opposed to r» 0.205 between SAT-M and P). The 
relatively low correlations between the SAT-M score and grade earned is consistent with 
Bridgeman (1982) but the predictive validity coefficients seem quite lower than found in 
the McLoughlin (2000) study where the correlations were for the score earned on the EAS 
and grade earned in College Algebra (r« 0.332), the score earned on the LAS and grade 
earned in Pre-calculus (f « 0.334), and between the score earned on the CR and grade 
earned in Calculus I (? « 0.433) which were moderate and did not seem to conflict with 
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previous studies (e.g.: Bridgeman 1980, 1981; Suddick & Collins, 1984; Dwinnell, 1985; 
and Meyer, Woodard, and Suddick, 1994). 

Let us return to consider the models for the DTMS as opposed to the SAT-M. Note 
that the slope of the regression equations is quite small for all the models but for each 
model of a test of the DTMS the slope is greater than that for the SAT-M: 

A = 1.504 + 0.039E for the EAS of the DTMS versus A = 0.241 + 0.004S for the SAT-M; 
P = 1.030 + 0.0841 for the IAS of the DTMS versus P = 0.695 + 0.003S for the SAT-M; 
and, C = 1.670 +0.050R for the CR of the DTMS versus C = -0.519 + 0.005S for the 
SAT-M. So, it seems that the DTMS might be a better predictor of grade earned in the 
respective courses (this cannot be stated with any confidence until the comparative part of 
the study is finished when the researchers gain access to the 1999 SAT-M data and 
compleat the study as originally designed). 

That the DTMS seems to be a better predictor is not surprising given that it was 
designed as a placement tool. Further, since the levels of the DTMS correspond with the 
material contained within the courses at the college that one might gain exemption from, 
Basic Mathematics, College Algebra, and Pre-calculus, it is logical that there would be 
some correspondence between the score and grades earned in the courses. However, the 
predictive validity coefficients were low. Nonetheless, the DTMS does seem better suited 
than the SAT-M on a content level but surprisingly the SAT-M performed as well in many 
instances as the DTMS. Nonetheless, the predictive coefficients found for the SAT-M for 
the courses was lower than Bridgeman & Wendler (1989) but they corrected for range 
restriction, “after correcting for the considerable range restriction that may occur when 
within-course scores are analyzed, coefficients were typically only in the mid 0.30s.” 

Also, there is a stability of scores earned on each of the tests of the DTMS which allow 
for comparatives from one year to the next (see graph 1 for an example) beside the fact that 
the test itself remains constant form year-to-year. 
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Graph 1: EAS of the DTMS Mean Grades 1989 - 1998. 

Indeed, there are more responses required of each examinee (95 questions for the 
DTMS as opposed to 60 for the SAT-M) so more information is gleaned from the DTMS 
than SAT-M. Further, there is a correspondence as noted between material in the Basic 
Mathematics course and the EAS, material in the College Algebra course and the IAS, 
and material in the Pre-calculus course and the CR; whereas, such correspondence does 
exist between the SAT-M and the courses but the score earned on the SAT-M may or may 
not reflect the content per course (there is no way to know since sub-scores are not reported 
and different examinees can earn equal scores whilst not having the similar content 
competency). Nonetheless, this might account for the higher correlation between SAT-M 
and grade earned in Calculus I and seems to support the findings of Bridgeman & Wendler 
(1989) that the SAT-M seems better suited for use in placement decisions for Calculus I 
but not for the lower-level courses. 

There existed a subtle restriction of range for each of the models for the regressors 
(score on EAS, IAS, CR, or SAT-M). For example, for the model A = 1.504 + 0.039E for 
the EAS of the DTMS there was a restriction of scores such that it was skewed negatively. 
Indeed, for high scores on the EAS there were not many grades for College Algebra since 
those who passed the EAS along with the IAS or CR or both were exempted from the 
College Algebra course, and therefore placed in either Pre-calculus or Calculus. Similar 
problems with restriction of range were true for the IAS with grades in Pre-calculus and 
CR scores and grades in Calculus I. The problem was compounded with the SAT-M 
scores since that was but one test used in a four-step placement process which though not 
used in 1998 will be used in 1999. Further, as previously noted there seemed to be a 
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truncation for the grades earned in the courses which caused a negative skew for the grades 
in College Algebra, Pre-calculus, and Calculus I. 

The truncation affected two of the assumptions necessary for inferences from the 
regression. It did not affect the assumption of homogeneity of variance; but it did affect the 
assumption of normality - - there was skewness to the distribution of grades earned in each 
of the course and with the scores on the tests. However, since regression analysis is robust 
to mild violations of the normality assumption, the assumption of normality was tenable. 
The truncation had an effect on the assumption of linearity and on the strength of the 
relationship between the score earned on the EAS, IAS, CR, or SAT-M and the grade 
earned in respective course. This truncation had the effect of lowering the estimate of R 
found in the study. That may explain why such small psychometric indices were found for 
the EAS, IAS, and the CR in predicting the grade earned in College Algebra, Pre-calculus, 
and Calculus I (respectively) and for the SAT-M predicting the grade earned in College 
Algebra, Pre-calculus, and Calculus I. Albeit this result is exploratory and conditional; but, 
it does make for an interesting comparison with past studies of the DTMS. This is because 
all of the studies previously conducted differed in that the samples were not truncated as in 
this study. 

For example, Bridgeman (1982) found the correlation between the grade earned in an 
Elementary Algebra course and the score earned on the EAS for a sample of 73 subjects at 
a small private 2-year college in the South was 0.46. Further, Bridgeman found the 
correlation between the grade earned in an Elementary Algebra course and the score earned 
on the EAS for a sample of 198 subjects at a large public 4-year college in the South was 
0.47. For the revised DTMS, Meyer, Woodard, and Suddick (1994) found the correlation 
between the grade earned in an Elementary Mathematics Concepts and Structures course 
and the score earned on the EAS for a sample of 60 subjects at a medium sized public 4- 
year college in the Midwest was 0.49. 

It would seem if truncation had not been the case in this study, then the resulting 
psychometric index might be more similar to psychometric indices found in previous 
studies. However, it may not because the statement is conditional. Further, the SAT-M part 
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of the study for students entering in 1999 has not been executed so the findings are 
prefatory at best. 

The aforementioned problems seem to indicate that a different method of analysis 
might or should be employed to determine the comparative utility of the DTMS and SAT- 
M as placement tools. The researchers have decided to use relative operating characteristic 
(ROC) analysis in a follow-up study to the present study by re-analysing the data using 
ROC analysis. There should be less sensitivity to truncation and restriction of range but 
some of the decision definitions will have to be revised. 

Perhaps the most interesting component of the study was not the regression equations, 
correlations, or any other parametric statistical portion of the study but the effect of the 
change of policy on the placement schema at the college (which needs to be studied 
further). Consider the placement that would have been the case had the DTMS not been 
used and instead the SAT-M had been used to place students in 1998 (see tables 4, 5, and 
6). Table 4 shows the placement of the sample that would have been had the SAT-M 
scores been used for the 186 subjects. Nonetheless, table 5 shows the actual placement for 
the 186 subject; and, table 6 illustrates a comparison between the two. Note that for the 
DTMS more students are placed in Basic Mathematics than would have been the case (and 
might be the case for the 1999 subjects) for the SAT-M. There would have been a less 
pronounced shift amongst students from the intermediate categories and there is a 
noteworthy shift also for those placed in Calculus I. 



SAT-M 




Basic Math 
Math 090 


College Algebra 
Math 100 


Pre-calculus 
Math 120 


Calculus I 
Math 161 


Number* 

(11 non-reported) 


9 


93 


44 


28 



Table 4: Placement of the sample that would have been had the SAT-M been used in 1998. 



DTMS 




Basic Math 
Math 090 


College Algebra 
Math 100 


Pre-calculus 
Math 120 


Calculus I 
Math 161 
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Number 


48 


71 


40 


26 



Table 5: Actual placement of the sample by the EAS, IAS , and CR of the DTMS in 1998. 







SAT-M 












Basic Math 
Math 090 


College 
Algebra 
Math 100 


Pre-Calculus 
Math 120 


Calculus I 
Math 161 


Total 




Failed EAS 
(Math 090) 


7 


38 


1 


0 


46 


DTMS 


Passed EAS 
Failed IAS 
(Math 100) 


2 


41 


22 


1 


68 




Passed IAS 
Failed CR 
(Math 120) 


0 


11 


15 


12 


38 




Passed EAS, 
IAS, & CR 
(Math 161) 


0 


3 


6 


15 


24 




Total 


9 


93 


44 


28 


174 



Table 5: Comparison of the actual placement of the sample by the EAS , IAS , and CR of the DTMS 
and of the sample that would have been had the SAT-M been used in 1998. 
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This seems to indicate that there was a shift of students such that less were placed in 
the remedial class and more into the courses that allowed a student to earn college credit 
and satisfy core curriculum requirements for mathematics. 

According the scoring guide for the SAT-M (2004) of the ETS the score of 410 
roughly corresponds with about 15 of 60 questions correct on the SAT-M, the score of 540 
roughly corresponds with about 34 of 60 questions correct on the SAT-M, and the score of 
600 roughly corresponds with about 42 of 60 questions correct on the SAT-M. The ETS 
notes that the scores are not predictors of grades in individual classes, the test is not content 
valid for placement purposes, and is only a predictor of freshman grade point average and 
should only be considered as one factor amongst a host of factors for admittance to college. 
The cut scores that are used on the SAT-M used to place students in 1999 to present at the 
college approximately correspond to 25% correct for the 410 score, 56 2 /3% for the 540 
score, and 70% for the score of 600. Note that the content areas are not delineated in these 
scores (the approximately 30% of the 60 questions covering Arithmetic (Number and 
Operations), the approximately 30% of the 60 questions covering Algebra and Functions, 
the approximately 30% of the 60 questions covering Geometry and Measurement, and the 
approximately 10% covering Data Analysis, Statistics, and Probability). 

Use of the SAT-M scores for a decision of placement into Calculus I does not seem 
problematic given that the cut scores that are used on the EAS, IAS, and CR are 70% for 
all of the tests. Nonetheless, the lack of delineation of sub-scores for each of the four 
content areas covered on the SAT-M might lead to a less reliable decision on placement of 
an entering freshman into mathematics courses. So, it seems intuitively that there is some 
correspondence in the placement schema between the SAT-M and DTMS for placement 
into Calculus I. However, for placement into Basic Mathematics, College Algebra, and 
Pre-calculus there seems to be little if any apparent connection. More research is needed to 
build evidence as to the advisability of use of the SAT-M scores as a placement tool as 
opposed to the DTMS. 

It should be noted that the use of SAT-M scores is more facile for the administrators, 
faculty, and students at the college. For placement with the use of the DTMS, the faculty 
was required to be proctors for the test during the week before classes commenced where 
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entering freshmen arrived at the campus for new student orientation. The scores were 
compiled using a scantron machine, but the scores were then manually recorded. No such 
need exists for the SAT-M since the scores are reported by ETS to the college (at the 
direction of individual applicants for admission to the college). 

The implication of the above result suggests that use of the EAS, LAS, and CR of the 
DTMS to identify students with deficient skills for college mathematics course work seems 
viable so that appropriate remediation can be effected for said students; but, the use of the 
SAT-M seems as suited since both demonstrate poor predictive validity for the data 
contained herein (if one were to make such a ‘dubious’ recommendation on the basis of the 
results contained herein). The practical difference of use of the DTMS scores as opposed to 
the SAT-M scores seems to be constrained to the number of students placed in the remedial 
course. There seems to be little evidence of the effect of remediation on a student’s 
performance in College Algebra; but, it should be noted that this was not a primary focus 
of this study. The inability to finish the study and compare the data results from this part of 
the study with data from entering freshmen in 1999 is regrettable and frustrating. 

Another comment is appropriate at this point of the discussion. Recall the 
mathematics department’s placement was based on the results of a student's performance 
on three (EAS, IAS, and CR) of the four components of the DTMS through 1998. In 1999, 
the placement scheme was changed to placement in freshman mathematics courses based 
on results from the SAT-M. The placement scheme was supposed to be placement in 
freshman mathematics courses was to be based on results from the SATII-Mathematics 
subject test. The policy was reported to the faculty of the Mathematics Department at the 
college by the Office of Academic Affairs and Office of Admissions; but, the placement 
scheme was changed to placement in freshman mathematics courses based on results from 
the SAT-M and was still the case for entering freshmen in fall of 2003. It is interesting to 
note that the faculty communicated with the Academic Affairs Office and inquired why the 
scheme changed; but, received no reply (for four years running). 

Nonetheless, policy questions aside, this study investigated the predictive validity of 
the DTMS as opposed to the predictive validity of the SAT-M in a population composed 
almost exclusively of African-American males which is interesting in and of itself. The 
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evidence found in this study somewhat adds to the body of evidence on the predictive 
power of the EAS, IAS, and CR of the DTMS as a tool in minimum competency testing. 
The evidence seems neutral (at this stage of the research) as to the use of the EAS, LAS, 
and CR of the DTMS as a placement tool as opposed to the SAT-M. Clearly more research 
is needed into the use of the DTMS and the SAT-M as placement tools (for this and other 
populations). As with any study, there are evidently more questions raised by the research 
than questions answered by the results. 
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